Recently, the intelligence community has moved aggressively to develop new IT infrastructures that make better use of their domain expertise to counter terrorism. These infrastructures include sophisticated modeling tools which are being deployed in computer-based collaboration environments to facilitate information exchange so that experts can incrementally improve their models as new information becomes available. While these infrastructures have the potential to improve the quality of intelligence produced by expert panels, and increase the speed at which it is produced, much of this potential has yet to be realized. This is mostly due to the fact that there is little effort to provide useful metrics for validating the quality of experts and the models they produce. Moreover, collaboration facilities mostly remain under-utilized and thus ultimately prove ineffective, largely due to the current lack of integration between information analysis/modeling tools and collaboration tools. This makes using collaboration tools “extra work” rather than routine, and often leads to undesirable situations where analysts are mostly unaware of each other's work, knowledge, and expertise. Hence, there is little motivation for them to proactively seek advice, initiate communication and collaborate.
During the last two decades there has been a significant increase in the research and development in the field of Computer Supported Cooperative Work (CSCW). Enthusiasm over the potential of collaboration technologies has caused some to deploy these as a means to improve knowledge creation and management in their work environment. For example, the Groove system was selected as a collaboration environment for a significant DARPA funded R&D program. The Groove system was developed by Groove Networks, Inc. (recently acquired by Microsoft, Corp.) and is described in Groove Virtual Office: Product Backgrounder (2005). As more of this type of “collaboration ware” becomes deployed in mission critical applications it becomes increasingly important not only for it to be used, but used effectively.
CSCW research has tended to focus on the production of new collaboration tools without concurrently developing new technologies to motivate their use. Some of this motivation might be provided by giving users greater insight into how knowledge is distributed within their work environments along with new communications interfaces, which, based on this insight, facilitate interactions between those who possess and those who need knowledge. This perspective is a departure from many process-based collaboration approaches designed to enforce prescribed work flows. Rather, a more flexible and informal knowledge-based approach is needed where collaboration grows out of a heightened awareness of who knows what. Of course, this approach will only gain acceptance by users if there is hope that, by collaborating with others, higher-quality knowledge will be produced, and there is also a way to validate the process whereby this knowledge is produced.
One of the collaborative modeling tools is the SIAM modeling tool. The SIAM modeling tool is described by Julie Rosen and Wayne Smith in “Influence Modeling for Strategic Planning: A Structured Approach to Information Operations,” Phalanx, vol. 33, No. 4, December 2000. SIAM is a decision support tool for collaborative Influence Net (or INET) modeling. INET modeling encourages panelists to create “influence nodes.” These influence nodes depict events that are part of cause-effect relationships within the situation under investigation. Panelists also create “influence links” between cause and effect that graphically illustrate the causal relation between the connected pair of events. This cause-effect relationship can be either reinforcing or reversing, as identified by the link “terminator”—an arrowhead or a circle. The resulting graphical illustration is called the Influence Net's “topology.” FIG. 1 is an influence network created with the SIAM modeling tool. Each node, such as nodes 100a and 100b, in FIG. 1 is an event and each link, such as link 110, connecting the nodes is an influence link representing the causal relation.
In the past, the SIAM tool was used primarily in face-to-face meetings to enter an INET “coaxed” from panelists by a moderator, one that best represented their consensus view. It would be desirable to enable collaborative construction of INETS by virtual panels, i.e., panels of experts that may be separated in time and space. Additionally, it would be desirable to be able to impose greater scientific rigor on the modeling process by identifying biases amongst panelists, qualifying panelists, deriving valid consensus models, and facilitating incremental improvement in models through further collaboration among panelists based on their level of knowledge and experience.
Consensus-based knowledge validation is useful when there is little time or money to conduct large-scale surveys across a large number of experts or a lack of quantitative data or practical impossibility make it difficult to conduct experiments necessary to produce data. Such an approach may also be necessary to avoid revealing intelligence concerns or targets. In “The ‘Emergent’ Semantic Web: A Consensus Approach to Deriving Semantic Knowledge on the Web,” the authors presented a formal model for deriving consensus from response data measured on a nominal scale, e.g., TRUE/FALSE or multiple-choice. This work did not provide a formal model for deriving consensus from response data measured on ordinal, interval or ratio scales.
Thus, it is an object of the present invention to provide consensus-based knowledge validation and analysis method and system that processes information acquired from human collaborators, representing diverse domains, such as information acquired through the human-machine interface available with the SIAM™ influence network (or INET) modeling tool. Such a method and system would support the derivation of consensus knowledge from which relevant, credible changes to knowledge corpora are detected, provide metrics to validate the derived knowledge and competency of human collaborators, and route new evidence to those whose assumptions are either supported or challenged by it. Unlike previous approaches that support process-based collaboration, i.e., interactions between experts based on organizational relationships, it would be desirable to leverage knowledge and its distribution among panelists to motivate use of available collaboration tools, i.e., knowledge-based collaboration, and the formation of advice networks. Furthermore, it is desirable to have a system that yields best answers based on responses of experts weighted by their respective competencies.
Furthermore, it would be desirable to provide such consensus analysis services through the Internet or virtual private network (“VPN”) as a Web Service (“WS’). Coupled with XML schemas for data input such services would be available to a wide variety of information analysis and modeling tools, even those that run on different software and hardware platforms. Furthermore, it would be desirable to provide for Java-based clients, validations and analysis results as Java objects.
It is unreasonable to assume that the same input data model would satisfy the data processing requirements of all possible modeling tools. At the same time, support for new modeling tools should not disrupt use of the method or require existing tools to change the way users access and use the knowledge validation service. Moreover, with the wide availability of computer-based collaboration tools that exists today, the method of the present invention does not seek to implement its own collaboration tools. Most groups already have collaboration tools that their members prefer or are required to use by policy. Since these tools are often designed or tailored to meet specific requirements of collaboration groups, it is unreasonable, and even unproductive, to impose an additional set of generic tools on collaborators. Ideally, the consensus-based knowledge validation and analysis tool should provide a collaboration interface through which users can easily access consensus analysis results and engage in collaboration on an as-needed basis using all (or any) of the existing collaboration tools in their IT environment. As the consensus-based knowledge validation and analysis tool cannot (or should not) have any prior knowledge of collaboration groups or their IT environments, this means that its collaboration interface should be able to dynamically discover what tools are deployed, then make them available to local users
It is desirable for such a knowledge-based model for collaboration to generate at least three supporting metrics: (1) a measure of the overall saliency of the knowledge domain to domain experts, a.k.a., subject matter experts (SMEs), (2) the level of domain expertise or “competence” for each SME with whom one might interact, and (3) the most probable set of “correct answers,” derived from the responses of each SME, i.e., the consensus view.
A single set of these metrics of a knowledge model gives a snapshot of knowledge distribution among subject matter experts (SMEs). It is also desirable to monitor, over time, the progress of consensus and knowledge building in the same group of SMEs. Thus, the consensus-based knowledge validation and analysis method and system should allow for analyzing a time series of knowledge models and generating visualizations and supporting metrics, which should include at least: (1) a measure of the overall knowledge variability amongst SMEs, (2) a measure of change in each SME's knowledge relative to peers from one period to the next, and (3) a measure of concordance from one knowledge model to the next.
The computer-implemented services should be scalable and extensible to a wide variety of collaborative modeling tools without requiring extensive customization, development and management overhead. Additionally, the consensus-based knowledge validation and analysis tool should easily and transparently integrate with collaboration tools that are locally available.